What operational steps are involved in adding a new node to an existing Qdrant cluster and rebalancing shards onto it?

Adding a node to an existing Qdrant cluster involves a sequence of steps: provision the node with the same configuration and version as the existing cluster, join it to the cluster's Raft group, wait for the cluster to recognize it, then trigger the placement or rebalancing logic to move some shard replicas onto the new node, and verify the transfer. The exact mechanism depends on the version: in some versions, the cluster automatically distributes shards when a new node joins; in others, the operator must trigger a rebalance. The transfer itself is a data migration: the new node receives a snapshot or a stream of the shard's data from a peer, applies it, and then joins the replication stream. The shard's primary continues to serve traffic during the transfer, so there is no downtime for that shard, but the transfer consumes network bandwidth and disk I/O on both the source and the destination node, which can affect query latency. The process must be done one shard at a time, or in small batches, to avoid saturating the network and causing latency spikes.

The mechanism that makes this safe is that shard replicas are independent: moving one replica of a shard does not affect the shard's availability, because the primary and the other replicas continue to serve. The cluster's metadata (maintained through Raft) tracks the placement, and the rebalance updates the placement atomically. The new node's replica is only promoted to primary if the primary fails, so the rebalance does not change the write path. The transfer is bounded by the size of the shard and the network bandwidth between the source and the destination. For a large shard, this can take minutes to hours. During the transfer, the source node's query latency may increase because of the I/O and network load, so the transfer should be scheduled during a low-traffic period if possible. After the transfer, the new node's replica is fully in sync and can serve reads, which increases the cluster's capacity. The rebalance is complete when the cluster's placement is balanced, which may require moving multiple replicas. The whole process should be monitored for progress, latency impact, and errors.

The trade-off is between adding capacity quickly and avoiding latency impact. A fast rebalance moves many replicas at once, which saturates the network and causes latency spikes. A slow rebalance moves one replica at a time, which is safe but takes longer. The right pace depends on the latency SLO and the size of the shards. The common mistakes are: (1) joining a node with a different version or configuration, which can cause compatibility issues; (2) triggering a full rebalance during peak traffic, which causes latency spikes; (3) not monitoring the transfer progress, so a stuck transfer is not noticed; (4) moving too many replicas at once, which saturates the network; (5) not verifying the placement after the rebalance, so some shards are still unbalanced. Version note: the cluster join and rebalance APIs have changed across Qdrant releases. In some versions, the rebalance is automatic; in others, it is manual. The exact API for adding a node and triggering a rebalance may differ. Verify the process on your version before performing it in production.

Version-dependent: the cluster join and rebalance APIs and the automatic placement behavior have changed across Qdrant releases. In older versions, the rebalance was more manual; in newer versions, it is more automated. Qdrant Cloud manages the process. If you are self-hosting, verify the exact process for your version and test it in a staging cluster before doing it in production.